Stream Processing in the Cloud

Stock exchanges, sensor networks and other publish/subscribe systems need to deal with highvolume streams of real-time data. Especially financial data has to be processed with low latency in order to cater for high-frequency trading algorithms. In order to deal with the large amounts of incoming data, the stream processing task has to be distributed. Traditionally, distributed stream processing systems balanced their load over a static number of nodes using operator placement or pipelining. In this report we propose a novel way of doing stream processing by exploiting scalable cluster architectures as provided by IaaS/cloud solutions such as Amazon’s EC2. We show how to implement a cloud-centric stream processor based on the MapReduce framework. We will then design a load balancing algorithm which allows a local stream processor to request additional resources from the cloud when its capacity to handle the input stream becomes insufficient .